Skip to content

Rediscover a stale payload policy on 413 payload_too_large and resend the completion once - #360

Merged
rmcdaniel merged 5 commits into
mainfrom
general/payload-policy-refresh
Nov 21, 2026
Merged

rmcdaniel merged 5 commits into
mainfrom
general/payload-policy-refresh

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Python counterpart of the PHP finding: a stale payload policy that leads to a 413 payload_too_large.

Is Python affected? Yes. It does not crash, but it loses the result.

  • The cached policy: Client._runtime_external_payload_transport (src/durable_workflow/client.py, around line 2435 on main) resolves namespace.external_payload_storage from cluster discovery once. That includes threshold_bytes, which is native Server's payload ceiling when the threshold is absent (rust/src/runtime/payload_discovery.rs::threshold). The result is cached in _runtime_external_payload_transport_cache, and _externalize_runtime_payloads keeps a payload inline when len(data) <= transport.threshold_bytes.
  • The refusal: native refuses an oversized worker request before processing with 413 {"reason":"payload_too_large","limit":N} (rust/src/runtime/payload_limits.rs::enforce).
  • On main:
    • An activity completion that gets that 413 is not transient. _deliver_activity_outcome raises, and _run_activity_task_body logs failed to complete activity task and returns complete_error.
    • The worker keeps running, unlike PHP, but the computed result is lost. The attempt fails later as a lost lease.
    • A workflow-task completion with limit_kind: workflow_task_completion_bytes fails the run straight away as a structural limit, even when only the cached policy was stale.

Fix

On a 413 payload_too_large for an activity completion, failure report or workflow-task completion:

  1. Rediscover the payload policy with GET /api/cluster/info. Outages are retried with the shared backoff while the lease, or 60 s if the lease is unknown, allows. Held discovery is replaced only by a successful read.

  2. Drop the cached policy and re-resolve it.

  3. If it changed, resend once. The request is re-encoded, so payloads that the new threshold requires are externalised. The 413 is refused before processing, so the resend is safe.

  4. Some activity results still cannot be delivered: the policy did not change, the resend is still too large, there is no external storage, or the result is over the upload limit (ExternalPayloadOversized). The attempt is then reported as a non-retryable failure with the same wire shape as PHP and Rust:

    • failure.type = "ActivityResultTooLarge"
    • failure.class = "ActivityResultTooLarge"
    • non_retryable = true
    • a message naming the Server's limit

    The attempt no longer expires silently.

  5. An oversized workflow-task completion keeps the structural-limit run failure.

The worker never exits and never reruns the activity, so the computed result is kept and sent.

Tests (tests/test_payload_policy_refresh.py)

The tests use a fake runtime whose request limit and threshold are lowered mid-run, after the worker cached the 2 MiB policy.

Case Expected
300 KiB result after the limit drops to 100 KiB Refused once inline, then resent once externalised. One upload, completed, the activity ran once, no failure report, and the new policy is held.
Rediscovery hits three 503s Rides them out, then resends.
Still too large after rediscovery (threshold unchanged) A single send, then one non-retryable ActivityResultTooLarge failure naming the 100 KiB limit. The worker's client keeps serving requests.
No external storage Fails the same way, with no upload attempted.
Workflow-task completion 413 Resent once when the policy changed; the existing structural-limit failure when it did not.

Local results (Python 3.14):

  • pytest tests/ -m "not integration": 7905 passed, 3 skipped.
  • ruff: clean.
  • mypy: clean.
  • regression-corpus guard: clean.

🤖 Generated with Claude Code

The worker resolves the runtime external payload policy once. When the
Server lowered its limit mid-run, a result under the old threshold was
sent inline and refused 413 payload_too_large; the worker logged
"failed to complete" and the result was lost (the activity later failed
as a lost lease).

On that 413 for an activity completion, failure report or workflow-task
completion, rediscover the policy (cluster info, retried through outages
with the shared backoff while the lease lasts) and, if it changed, resend
once re-encoded, externalising what the new threshold requires. The 413
is refused before processing, so the resend is safe. If the policy did
not change or the resend is still too large, today's handling stands.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel added bug Something isn't working agent:general labels Nov 20, 2026
rmcdaniel and others added 4 commits November 20, 2026 11:37
…arge

When the result is still too large after rediscovery, there is no
external storage to offload to, or it is over the upload limit, the
attempt used to expire silently. Report it as a non-retryable failure
with failure.type and failure.class "ActivityResultTooLarge" (the same
wire shape as PHP and Rust) and a message naming the Server's limit;
the worker keeps running.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
No behaviour change: read the refusal's limit without a redundant type
check, move the rediscovery loop into _rediscover_cluster_info (cognitive
complexity), log its final failure with log.exception, and split a
composite test assertion.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel rmcdaniel left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the coordinated worker payload-limit fix against the PHP #451, Python #360 and Rust #306 changes. PHP, Python and Rust recover only from the Server's typed size refusal. The worker can rediscover the policy and re-encode the result without re-running the activity. A result that still cannot fit is reported with type and class ActivityResultTooLarge and non_retryable: true, and worker dispatch remains available. A discovery outage keeps the attempt eligible for recovery instead of recording a permanent size failure.

The target-branch and language CI jobs passed for all three reviewed heads. Python also passed the local focused transport suite, and Rust passed the 17 worker-outage tests. The release qualification and published-package checks follow the merges.

@rmcdaniel
rmcdaniel merged commit 9a02f2f into main Nov 21, 2026
25 checks passed
@rmcdaniel
rmcdaniel deleted the general/payload-policy-refresh branch November 21, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant